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Executive Summary 


NOTICE: This is an interim report of a more comprehensive evaluation 
study. As such, the report is not intended to present summative conclusions 
about the efficacy or outcomes of the program under study. The intended 
purpose of this report is to present stakeholder feedback for program staff to 
consider as they make implementation decisions regarding the program. 


The Educator Evaluation Pilot Project Midyear Survey gathered teacher feedback 
about (a) how well various components of the new evaluation system are being implement- 
ed; (b) how much time, on average, is required to complete each component of the system; 
(c) how much each component contributes to professional growth; (d) perceptions and atti- 
tudes about the pilot; (e) the level and quality of support provided to teachers for the pur- 
pose of enabling them to participate in and implement the new system; and (f) the most 
beneficial and challenging aspects of the evaluation system. 


The Office of Research sent an e-mail message of survey invitation with a link to the 
online midyear survey on January 17th, 2012 to all teachers whose names and e-mail ad- 
dresses appeared on lists vetted by their principals (n = 765). After the initial invitation, 
three e-mail survey reminders were sent, spaced approximately 10 days apart leading up to 
the final deadline (February 20th, 2012). A total of 421 teachers from all 25 pilot schools 
completed the survey. This is a response rate of 55%, which means we can have 95% confi- 
dence (+3.2% margin of error) that the results of the survey are representative of the larger 
population. 


Findings 
Professional development, support and technology 


Two thirds of all respondents (67%) indicated they had received adequate training to 
participate in the new evaluation system. However, a little over 30% of respondents indicat- 
ed they did not attend one of the six professional development sessions on the new evalua- 
tion system, which took place in August 2011. 


The vast majority of respondents indicated they received adequate evaluation-related 
support from school administrators (74%) and that the evaluation-related feedback they had 
received from school administrators had been constructive and beneficial (67%). 


Although half of all respondents (79 out of 158) who encountered technology issues 
related to the pilot indicated the issues had been adequately addressed, the remaining half 
indicated they were still encountering technology issues. 


Implementation: fidelity 


Responses indicate that the system component with the highest fidelity of implemen- 
tation thus far is the student learning goal process (Table 1). Overall, about 96% of respond- 
ents, regardless of progression level, indicated that they had established goals and identified 
strategies, measures, and evidence. 


Executive Summary 


Approximately 13% of respondents on the initial and intermediate progressions indi- 
cated that school administrators had not conducted a classroom observation. 


About 80% of teachers on the initial and intermediate progressions indicated that 
they have had a conference regarding the student learning goals they established, and three 
fourths (74%) indicated that a postobservation conference had taken place. Less than two 
thirds of teachers (64%) on the advanced progression have had a conference with school 
administrators to discuss their self-assessment and student learning goals. 


Table 1. Percentage of Respondents Who Have 
Completed System Components 


Follow-up 
Component Completed conference 
Student learning goals 
(initial and intermediate) 95.9% 80.0% 
Observation (initial and 
intermediate) 87.1% 74.3% 
Self-assessment (advanced) 92.1% 
Student learning goals 64.3%* 
(advanced) 98.0% 


*Conference regarding self-assessment and student 
learning goals 


Implementation: time 


Approximately two thirds of all respondents indicated that it took them less than 60 
minutes to establish student learning goals and identify strategies, measures, and evidence. 
The percentages are slightly lower for teachers on the initial and intermediate progression 
(62%), compared to teachers on the advanced progressions (68%, Figure 2). 


Based on participant responses, the majority of student learning goals conferences 
took 30 minutes or less. The percentages of those requiring this brief amount of time are 
slightly higher for advanced teachers at 85%, compared to those in the initial and intermedi- 
ate progressions combined at 72% (Figure 2). Overall, 94% of teachers in the initial and in- 
termediate progressions and 98% of those in the advanced progression indicated that 
conferences lasted less than 60 minutes (Figure 2). 


About half of respondents (48%) on the advanced progressions completed their self- 
assessment in 30 minutes or less and 88% completed it in 60 minutes or less (Figure 2). 
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Amount of Time it Took Respondents to Complete Various Components of the System 


Figure 2. 


Teachers on Initial and Intermediate 
Progressions 


Teachers on Advanced Progressions 


Establish student learning goals 


@ Establish student learning goals ™ Conference about student leaming goals and self- 


‘ assessment 
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Figure 1. 

At this point, the compo- 
nent of the system that re- 
spondents viewed as_ having 
contributed the most to their 


Teachers on Initial and Intermediate Progressions 


Developing student learning goals 


Conference to discuss learning goals 
Classroom observation 
Post-observation conference 


Process of compiling additional 
evidence 
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© Percentage of respondents indicating at least a moderate impact 


Teachers on Advanced Progression 


Developing student learning goals 
Conference to discuss learning goals 
Self-assessment 


Conference regarding self-assessment 


Process of compiling additional 
evidence 


0% 20% 40% 60% 


m Percentage of respondents inidcating at least a moderate impact 


professional growth is the pro- 
cess of developing student 
learning goals (Figure 1). The 
percentage of respondents with 
this view is slightly higher 
among teachers on the ad- 
vanced progression (79%) com- 
pared to those on the initial and 
intermediate progressions 
(74%). 


Both groups of respondents 
ranked as second—in terms of 
their contribution to their pro- 
fessional growth—their confer- 
ences [with school administra- 
tors] to discuss student learning 
goals. Teachers in the initial 
and intermediate progres-sions 
rated these conferences higher 
compared to those in the ad- 
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Executive Summary 


vanced progression (72% and 64%, respectively). 


The process of compiling evidence was viewed as contributing the least to teachers’ 
professional growth. This was particularly the case for teachers in the advanced progression 
(47% compared to 61%). 


Teachers’ attitudes and perceptions about the system 


More than 60% of respondents indicated that they believe that (a) they play an active 
role in their own evaluation (71%); (b) the new evaluation system promotes professional 
growth (63%); (c) the new system clarifies what is expected from teachers (68%); and (d) the 
district/school has provided enough time for them to collaborate with other teachers in their 
school (65%) (Figure 3). 


Approximately half or more of respondents indicated that they believe the new eval- 
uation system is empowering to teachers (50%) and that it is fair to all teachers regardless of 
tenure, role, and so forth (55%). 


Figure 3. Respondents’ Perception and Attitude about the Pilot 


Play an active role in evaluation. 

System clarifies what is expected. 
Have been provided enough time for... 

System promotes professional growth. 

System is fair to all teachers 

System is empowering to teachers. 


System is better than the previous one. 


Has had a positive impact on student... 


Faculty have a positive attitude about the system. 


Strongly Agree & Agree — Disagree & Strongly Disagree Too early to tell &| don't know 


Recommendations 


It is worth noting here that feedback from participants is gathered at the halfway 
point of the first year implementation of the pilot project. Therefore, data from this interim 
evaluation report should be taken not to pass judgment on the merit of the system but rather 
to identify the strengths and weakness of the system during the early phases of implementa- 
tion. The information should be used diagnostically as a “temperature check” to provide ad- 
ditional clarifications and support to schools participating in the pilot project. The feedback 
from the experiences of pilot schools could also be utilized to better inform the scaling-up 


process in coming years. With this caveat in mind, the following recommendations are pro- 
vided: 


e Consider establishing an ongoing monitoring protocol to identify the number of 
teachers from each school who may not be able to attend training sessions in the be- 
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ginning of the school year and encourage RESA staff and/or school leadership teams 
to provide supplemental training on an ongoing basis. 


e The 25 pilot schools would have benefited greatly if a single individual had been des- 
ignated from the Office of Information Systems at the WVDE to serve as a contact 
person and resolve technical issues in a timely manner for schools and districts. Go- 
ing forward, and with the addition of more than 100 schools in the pilot, the same 
option is not possible. Consider providing extensive training, specifically on the 
online system, to individuals either at the RESA or district level to serve as contact 
persons for their schools. 


e Consider making WVEIS on the Web accessible to educators outside of the school 
building. This may allow teachers to devote more time to various components of the 
system outside of the school day and avoid some technical difficulties resulting from 
limited bandwidth at the school level and, in the long-term, may lead to better overall 
quality implementation. 


e Consider making the self-assessment instrument available for the less experienced 
teachers and encouraging them to utilize the process for their own purposes. One of 
the most beneficial components of the system identified by respondents in the ad- 
vanced progression is the process of completing a self-assessment. 


e Consider providing onsite technical assistance to provide clarification on the process 
of setting student learning goals. All respondents identified this component as the 
most beneficial part of the system. Educators believe the process helps them ap- 
proach their teaching more systematically by helping them focus on areas of needed 
improvement for their students while it holds them accountable to the goals they 
have set for themselves and their students. At the same time, however, feedback from 
respondents indicates that participants still consider this process to be the most chal- 
lenging part of the pilot project. 


e Consider elucidating further the process for compiling additional evidence. Over a 
third of respondents indicated that they need additional clarification on this aspect of 
the pilot project. 
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Introduction 


NOTICE: This is an interim report of a more comprehensive evaluation 
study. As such, the report is not intended to present summative conclusions 
about the efficacy or outcomes of the program under study. The intended 
purpose of this report is to present stakeholder feedback for program staff to 
consider as they make implementation decisions regarding the program. 


The Educator Evaluation Midyear Survey was designed by the Office of Research to 
be a formative evaluation tool for gathering teacher feedback about (a) how well the various 
components of the new evaluation system are being implemented; (b) how much time, on 
average, each component of the system requires to complete; (c) how much each component 
contributes to professional growth; (d) the level and quality of support provided to teachers 
for the purpose of enabling them to participate in and implement the new system; and (e) 
the most beneficial and challenging aspects of the evaluation system. 


Methods 


To assure a high level of response to the survey, staff in the Office of Professional 
Preparation (Certification) consulted master calendars for all 25 schools participating in the 
pilot, to obtain comprehensive lists of names and e-mail addresses for all professional staff 
at the schools. The lists were then sent to principals to verify their accuracy and make need- 
ed revisions. The principals sent the vetted lists of educator names, including a viable e-mail 
address for each, back to the Office of Professional Preparation. 


Using these lists, the Office of Research sent a letter of survey invitation with a link to 
the online midyear survey on January the 17th, 2012 to all teachers that appeared on the lists 
vetted by their principals (n= 765). After the initial invitation, we sent three survey remind- 
ers, Spaced approximately 10 days apart (January 26th, February 7th, and February 17th) 
leading up to the final deadline (February 20th, 2012). 


Results 


Respondents 


1.1. A total of 421 teachers completed to the survey. This is a response rate of 55.0%, which 
means we can have 95% confidence (+3.2% margin of error) that the results of the sur- 
vey are representative of the larger population. 


1.2. Educators from all 25 pilot schools participated in the survey. 


1.3. There were approximately twice as many respondents on the advanced progression (n= 
202, 48%) as there were teachers in the initial (n= 117, 27.8%) or intermediate pro- 
gressions (n= 102, 24.2%). 


1.4. Overall, 30.9% of all respondents indicated that they did not attend one of the six pro- 
fessional development sessions on the new evaluation system that took place in August 


2.1. 


2.2. 


Results 


2011. Percentages of respondents who did not attend any of the PD sessions were high- 
er for teachers on the initial progression, 36.8%, than those on the intermediate pro- 
gression (28.4%) or the advanced progression (28.7%). 


Technical Issues 


A little over a third of respondents (37.6%, n=158) indicated that they had encountered 
technological issues with the online system. Of those who had encountered technologi- 
cal issues, 79 respondents or 50.0% indicated that issues had been adequately ad- 
dressed. The remaining respondents indicated that technological issues had been 
addressed to some extent (41.1%) or had not been addressed (8.9%). 


Very preliminary: Comments indicate that the vast majority of technological prob- 
lems—initial and ongoing—involve logging on to the system and/or accessing data al- 
ready submitted in order to revise it. At this time, it is not clear if the issues are due to 
local internet access or WVEIS on the Web. Comments have been forwarded to the Of- 
fice of Information Systems. 


Implementation: Initial and Intermediate Progressions 


Student learning goals Table 2. How much time did it take to complete the 
Student Learning Goal document, e.g., 

3.1. Of 218 respondents, 209 or establishing goals and identifying strategies, 
95.9% indicated that they had es- measures, and evidence? 
tablished student learning goals. Answer options Response 

3.2. Most respondents (128 out of Less than 15 minutes 1.0% 
208, 61.5%) indicated that it took 46-309 minutes a ey 
them 60 minutes or less to com- ; 

4 31-45 minutes 16.8% 
plete the student learning goals 
document (Table 2). 46-60 minutes 32.2% 

61-90 minut 
3.3. Most respondents (168 out of Li laads 14.4% 
210, 80.0%) also indicated they 91-120 minutes 11.5% 
had a conference with their prin- 121-150 minutes 1.9% 
cipal or assistant principal to dis- 464-180 minutes ‘ae 
cuss student learning goals they 
Submitted, More than 3 hours 3.7% 
Note: The category with the highest percent of responses is 
3.4. Of those who had a conference haaed oe les y 


with their school leaders 121 out 


of 168 or 72.0% indicated that the conference took 30 minutes or less. 


Classroom observation 


3.6. 


When asked if they have had a postobservation conference with their princi- 
pal/assistant principal during the fall semester, 162 out of 218 or 74.3% of respondents 
said yes, 12.8% said no, and 12.8% said “[their] principal/assistant principal did not 
conduct an observation in [their] classroom in the fall semester.” 
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Results 


3.6. Of those who responded yes to the question above (#3.5), 161 out of 165 or 97.6% indi- 
cated that the conference took place within 10 days of the observation. 


2.7, 


3.8. 


3.9. 


4.1 


4.3 


Of those who have had a post- 
observation conference, 132 out of 
162 or 81.5% said the conference 
lasted 30 minutes or less (Table 


3). 


Of 162 respondents, 63 or 38.9% 
indicated that they completed an 
evidence form supporting class- 
room observation(s). Almost half 
of those who completed the form 
(47.7%) said they submitted it 
online. 


Besides completing an evidence 
form, 29 out of 63 (46.0%) indi- 
cated that they presented addi- 
tional evidence during’ the 
postobservation conference. On 
average, it took respondents 53.7 
minutes to compile the evidence 
they presented during the confer- 
ence (median= 37.5 minutes). 


Table 3. How long did the postobservation 
conference last? (Please provide an 
estimate.) Note: If you have had more than 
one conference in the fall semester please 
provide an average estimate. 


Answer options Response 
Less than 15 minutes 36.4% 
16-30 minutes 45.1% 
31-45 minutes 13.0% 
46-60 minutes 3.1% 
61-90 minutes 0.6% 
91-120 minutes 1.2% 
121-150 minutes 0.0% 
151-180 minutes 0.0% 
More than 3 hours 0.0% 


Note: The category with the highest percent of responses 
is shaded. 


Impact of System Components on Professional Growth for Teachers on Initial and 


Intermediate Progressions 


Respondents were asked to indicate the degree to which various evaluation compo- 
nents have contributed to their professional growth thus far. Approximately 214 teachers 
responded to five items in this section of the survey (see Table 4 for a breakdown of percent- 
ages in all response categories excluding not applicable responses). 


Overall, 157 or 74.4% of respondents indicated that the process of developing student 
learning goals had contributed to their professional growth at least moderately. About 
39.3% (n= 83) indicated that the process had contributed to their professional growth 


very much or extremely. 


Over two thirds (n= 141, 71.9%) of respondents indicated that conference with princi- 
pal/assistant principal to discuss learning goals had contributed to their professional 
growth at least moderately; with 82 or 41.8 % of respondents indicating that the pro- 
cess had contributed to their professional growth very much or extremely. 


Over two thirds (n= 142, 71.7%) of respondents also indicated that classroom observa- 
tion had contributed to their professional growth at least moderately; with 83 or 41.9% 
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4.4 


4.5 


Results 


of respondents indicating that the process had contributed to their professional growth 
very much or extremely. 


Approximately two thirds (n= 123, 68.0%) of respondents indicated that postobserva- 
tion conference with principal/assistant principal had contributed to their profes- 
sional growth at least moderately; with 77 or 42.5 % of respondents indicating that the 
process had contributed to their professional growth very much or extremely. 


A little under two thirds (n= 110, 61.1%) of respondents indicated that the process of 
compiling additional evidence had contributed to their professional growth at least 
moderately; with 55 or 30.6% of respondents indicating that the process had contrib- 
uted to their professional growth very much or extremely. 


Table 4. To what degree have the following evaluation components contributed to your professional 


growth thus far? (Select "not applicable" if tasks are not applicable to you.) 


Too early 

Items Not at all Slightly Moderately Very Extremely to tell 
Developing student learning 
goals 7.6% 11.8% 35.1% 28.4% 10.9% 6.2% 
Conference with 
principal/assistant principal to 
discuss learning goals 12.2% 12.2% 30.1% 31.6% 10.2% 3.6% 
Classroom observation 11.6% 13.1% 29.8% 31.3% 10.6% 3.5% 
The process of compiling 
additional evidence 15.6% 12.2% 30.6% 23.3% 7.2% 11.1% 
Postobservation conference 
with principal/assistant 

12.7% 13.3% 25.4% 32.0% 10.5% 6.1% 


principal 


Note: The response with the highest percent of respondents is shaded for each item. 


Implementation: Advanced Progression 


Self-assessment 


Bal, 


5.2. 


5.3. 


5a 


Of 203 respondents indicating they were on the advanced progression, 187 or 92.1% 
responded they had completed their self-assessment. Of those who had completed 
their self-assessment, 177 out of 189 or 93.7% of them indicated that they had submit- 
ted it online. 


Approximately 48% (90 out of 187) of respondents indicated that it took them 30 
minutes or less to complete their self-assessment (Table 5). 


Of 187 respondents who completed their self-assessment, 52 or 27.8% indicated that 
they had completed an evidence form supporting rating at the distinguished level. Ap- 
proximately 75.9% of these respondents indicated that they had submitted the form 
online. 


Besides completing an evidence form, 18 of 51 (35.3%) indicated that they presented 
additional evidence during the conference with their principal/assistant principal. On 
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Table 5. How long did it take you to 


complete the self-assessment? 


eee — respondent who indicated that he or she spent 5 
sane nine 10.2% hours and 15 minutes compiling evidence. Exclud- 
ateee minuees 38.0% ing that respondent, the average number of 
31-45 minutes 24.6% minutes drops down to 49. The median [the mid- 
46-60 minutes 15.5% dle] value is 45 minutes.) 

61-90 minutes 7.5% 

91-120 minutes 2.7% 

121-150 minutes 0.0% 

151-180 minutes 0.0% 

More than 3 hours 1.6% 


Note: The category with the highest percent 
of responses is shaded. 


Student learning goals 


5-5- 


5.6. 


5-7- 


5.8. 


Of 202 respondents, 198 or 98.0% 
indicated that they had established 
student learning goals. 


Approximately 68.2% of respondents 
(135 of 198) indicated that it took 
them 60 minutes or less to complete 
the student learning goal document 
(Table 6). 


Of 196 respondents, 126 or 64.3% in- 
dicated that they had a conference 
with their principal/assistant princi- 
pal about their self-assessment and 
student learning goals. 


Of those who had a conference with 
their school leaders, 107 of 126 or 
84.9% indicated that the conference 
took 30 minutes or less. 


average, it took respondents 61.5 minutes to com- 
pile the evidence they presented during the con- 
ference. (Note: the average is skewed due to one 


Table 6. How much time did it take to complete 
the Student Learning Goal document, 
e.g., establishing goals and identifying 
strategies, measures, and evidence? 


Answer options Response 
Less than 15 minutes 2.5% 
16-30 minutes 15.2% 
31-45 minutes 22.2% 
46-60 minutes 28.3% 
61-90 minutes 14.1% 
91-120 minutes 6.6% 
121-150 minutes 3.5% 
151-180 minutes 0.5% 
More than 3 hours 7.1% 


Note: The category with the highest percent of 
responses is shaded. 


Impact of System Components on Professional Growth for 


Teachers on the Advanced Progression 


Respondents were asked to indicate the degree to which various evaluation compo- 
nents had contributed to their professional growth thus far. Approximately 198 teachers re- 
sponded to five items in this section of the survey. Results are summarized below (see Table 
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7 for a further breakdown of percentages in all response categories excluding not applicable 
responses). 


6.1. Overall, 154 or 78.6% of respondents indicated that the process of developing student 
learning goals had contributed to their professional growth at least moderately. Of 
these respondents, 82 or 41.8% indicated that the process had contributed to their pro- 
fessional growth very much or extremely. 


6.2. Approximately two thirds (n= 98, 64.1%) of respondents, indicated that the conference 
with principal/assistant principal to discuss learning goals had contributed to their 
professional growth at least moderately; with 46 or 30.1% of respondents indicating 
that the process had contributed to their professional growth very much or extremely. 


6.3. Most respondents (n= 86, 62.5%) indicated that the process of completing a self- 
assessment had contributed to their professional growth at least moderately; with 51 
or 26.6% of respondents indicating that the process had contributed to their profes- 
sional growth very much or extremely. 


Table 7. To what degree have the following evaluation components contributed to your professional 
growth thus far? (Select "not applicable" if tasks are not applicable to you.) 


Too early to 
Item Not at all Slightly Moderately Very __ Extremely tell 
Self-assessment 12.0% 10.4% 35.9% 21.9% 4.7% 15.1% 
Conference with 18.7% 12.0% 30.7% 21.3% 4.7% 12.7% 
principal/assistant 
principal regarding self- 
assessment 
The process of compiling 19.2% 14.5% 27.9% 14.0% 4.7% 19.8% 
additional evidence for 
conference regarding self- 
assessment 
Developing student 6.6% 9.2% 36.7% 31.6% 10.2% 5.6% 
learning goals 
Conference with 11.8% 11.8% 34.0% 22.9% 7.2% 12.4% 


principal/assistant 
principal regarding student 
learning goals 


Note: The answer option with the highest percent of responses is shaded for each item. 


6.4 A little over half (n= 85, 56.7% of respondents indicated that the conference with prin- 
cipal/assistant principal to discuss self-assessment had contributed to their profes- 
sional growth at least moderately; with 39 or 26.0% of respondents indicating that the 
process had contributed to their professional growth very much or extremely. 


6.5 A little under half (n= 80, 46.5% of respondents indicated that the process of compil- 
ing additional evidence had contributed to their professional growth at least moder- 
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ately; with 32 or 18.6% of respondents indicating that the process had contributed to 
their professional growth very much or extremely. 


Perceptions and Attitude about the Pilot: All Teachers 


Respondents were asked to indicate the degree to which they agreed or disagreed 


with nine statements designed to gather data on their perceptions and attitudes about the 
new educator evaluation system. Approximately 412 teachers responded to the nine survey 
items. Results are summarized below (see Table 8 for a further breakdown of percentages in 
all response categories). 


71. 


yes 


7.3. 


7.4. 


7-3: 


More than 60% of respondents either strongly agreed or agreed with the following 
four statements: “Under the new evaluation system, I feel I play an active role in my 
own evaluation” (n= 293, 71.1%); “The new evaluation system promotes professional 
growth” (n= 257, 62.7%); “The new evaluation system clarifies what is expected from 
teachers” (n= 278, 67.8%); and “My district/school has provided enough time for me to 
collaborate with other teachers in my school” (n= 264, 64.5%). It is worth mentioning 
here that 118 or 28.9% of respondents either strongly disagreed or disagreed with the 
last statement. 


Approximately half or more of respondents either strongly agreed or agreed with the 
following two statements: “The new evaluation system is empowering to teachers” (n= 
205, 49.8%) and “The new evaluation system is fair to all teachers regardless of tenure, 
role, etc...” (n= 225, 54.7%). Approximately a third of respondents indicated that it was 
either too early to tell or that they don’t know in regards to these two survey items (n= 
141, 34.2% and n= 122, 29.7%, respectively). 

Of 411 respondents, 167 or 40.6% strongly agreed or agreed that “The new evaluation 
system is better than the previous one.” A similar percentage of respondents (n= 165, 
40.1%) indicated that it was either too early to tell or that they don’t know. The re- 
maining 79 or 19.2% either strongly disagreed or disagreed with the statement. 


Approximately half of respondents (n= 208, 50.7%) indicated that it was either too 
early to tell or that they don’t know whether “The new evaluation system has had a 
positive impact on student performance.” However, 139 or 33.9% of respondents either 
strongly agreed or agreed with the statement. 


In response to whether “faculty members at [their] school generally have a positive at- 
titude about the new evaluation system,” 156 or 37.9% indicated that it was either too 
early to tell or that they don’t know. A slightly lower percentage of respondents 
(33.7%, n= 139) either strongly agreed or agreed with the statement and 117 or 28.4% 
of respondents either strongly disagreed or disagreed with the statement. 
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Table 8. Respondent's Perceptions and Attitude about the Pilot 


Strongly Strongly Too early I don't 
Answer options Agree Agree Disagree disagree to tell know 
Under the new evaluation system, | 11.7% 59.5% 6.3% 1.9% 16.7% 3.9% 
feel | play an active role in my own 
evaluation. 
The new evaluation system is 7.3% 42.5% 12.4% 3.6% 26.9% 7.3% 
empowering to teachers. 
The new evaluation system is better 8.0% 32.6% 14.4% 4.9% 26.3% 13.9% 
than the previous one. 
The new evaluation system promotes 8.8% 53.9% 8.5% 3.2% 20.2% 5.4% 
professional growth. 
The new evaluation system is fair to all 7.3% 47.4% 10.7% 4.9% 19.0% 10.7% 
teachers regardless of tenure, role, 
etc... 
The new evaluation system has had a 6.1% 27.8% 12.4% 2.9% 43.9% 6.8% 
positive impact on student 
performance. 
Faculty members at my school 2.4% 31.3% 22.3% 6.1% 15.5% 22.3% 
generally have a positive attitude 
about the new evaluation system. 
The new evaluation system clarifies 7.8% 60.0% 12.4% 3.9% 9.3% 6.6% 
what is expected from teachers. 
My district/school has provided 9.3% 55.3% 18.8% 10.0% 4.6% 2.0% 


enough time for me to collaborate 
with other teachers in my school. 


Note: The category with the highest percent of responses is shaded. 


8.1. 


8.2. 


8.3. 


Support 


Slightly over two thirds of all respondents strongly agreed or agreed that “[They] have 
received adequate professional development to enable [them] to participate in the new 
evaluation system” (275 of 411 or 67.1%). 


Almost three quarters of all respondents strongly agreed or agreed that “[They] have 
received adequate evaluation-related support from school administrators” (302 of 407 
or 74.2%) and slightly over two thirds of respondents strongly agreed or agreed that 
“The evaluation-related feedback [they] have received from school administrators has 
been constructive and beneficial” (272 of 405 or 67.2%). 


Of 410 respondents, 243 or 59.3% strongly agreed or agreed that “Overall, the new 
evaluation system has been well implemented in [their] school.” A little over a quarter 
of respondents indicated that they don’t know or that it was too early to tell (n= 123, 
27.6%). 
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8.4. 


When asked which components of the pilot need further clarification, 165 of 371 or 
44.5% indicated that all components of the pilot had been well explained and they did 
not need further clarification. However, there seems to be a need for clarification with 
regard to the process for compiling additional evidence for self-assessment (142 of 371 
or 38.3%) and postobservation conference (136 of 371 or 36.7%) and, to a lesser extent, 
the student learning goal-setting process (92 of 371 or 24.8%). 


Open-Ended Responses 


Respondents were asked to comment on what they believed were the most positive 


and challenging aspects of the evaluation system. Of the 421 teachers who completed the 
survey, 179 or 42.5% provided comments regarding the most positive aspects of the system 
while 205 teachers or 48.7% provided comments regarding what they believe to be the most 
challenging aspects of the new evaluation system. 


Most positive aspects of the system 


9.1. 


9.2. 


9.3. 


9.4. 


9.5. 


Three dominant themes emerged from respondent comments regarding the most posi- 
tive aspects of the new evaluation system. Over half of all responses to this survey item 
fall under one of these three themes: self-reflection, empowerment, and goal-setting. 


Comments indicate that the new system provides opportunities for teachers to self- 
reflect and examine closely their own strengths and weakness so that they can work 
toward becoming more effective educators. The vast majority of those who commented 
on this positive aspect of the system were teachers in the advanced progression, who 
are required to complete a self-assessment as part of their evaluation. 


Another positive aspect of the new evaluation system identified by respondents was the 
sense of ownership that participants have about the evaluation process. Respondents 
believe they play an active role in their own evaluation. To some extent the process is 
self-guided because they can set goals for student learning that are specific to their 
content areas, as well as present evidence to demonstrate their effectiveness as educa- 
tors. Overall, comments indicated that teachers seem to appreciate the fact that they 
have some level of ownership in the process and they believe the system is empower- 
ing to teachers. 


Another feature of the new evaluation system identified by respondents as among its 
most positive aspects was the process of establishing student learning goals. The ma- 
jority of respondents simply identified this process as positive without any further 
elaboration. However, based on comments from those who did elaborate, it appears 
some teachers believe the process helps them approach their teaching more systemati- 
cally, at the same time it holds them accountable to the goals they have set for them- 
selves and their students. Respondents commented that they are now more focused on 
areas of needed improvement for their students. 


In addition to the three aspects of the system discussed above, respondents also identi- 
fied other positives, albeit less frequently, including opportunities for conferences with 
administrators, clearly stated expectations for teachers, teacher accountability, and a 
paperless evaluation process. 
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Most challenging aspects of the system 


9.6. Three dominant themes emerged from respondent comments regarding the most chal- 
lenging aspects of the new evaluation system. Approximately two thirds of all teachers 
who responded to this survey item identified one or more of the following three aspects 
as challenges they have faced thus far: time, the student learning goal process, and 
technical issues. 


9.7. Close to one third of respondents commented that finding the time to complete every- 
thing that is required under the new system has been very challenging. Comments 
from respondents regarding the time issue generally fall into three categories: (a) re- 
spondents feeling they already have “a lot on [our] plates”; (b) time issues directly re- 
lated to technical glitches they have encountered, which have made the process take 
longer than expected; and (c) the inability of teachers to access the online system out- 
side of the school building. In other words, teachers believe they do not have sufficient 
free time at school to be able to access WVEIS on the Web and complete the various 
requirements of the system online. 


9.8. The second challenge most frequently mentioned by respondents was participation in 
the student learning goal process. Although most respondents did not elaborate fur- 
ther, comments from those who did take the opportunity to explain their views of this 
issue indicate that this component of the evaluation system still needs further clarifica- 
tion. This may be due to the number of teachers who did not attend any of the profes- 
sional development sessions provided in July and August. 


9.9. The third most frequently mentioned barrier to effective implementation according to 
respondents is online technical issues, which have been adequately covered elsewhere 
in this report. 


9.10. Other less frequently mentioned challenges include the process of compiling additional 
evidence and completing the self-assessment, learning a new system and lack of ade- 
quate guidance, and lack of adequate training. 


Recommendations 


It is worth noting here that feedback from participants is gathered at the halfway 
point of the first year implementation of the pilot project. Therefore, data from this interim 
evaluation report should not be taken to pass judgment on the merit of the system but rather 
to identify the strengths and weakness of the system during the early phases of implementa- 
tion. The information should be used diagnostically as a “temperature check” to provide ad- 
ditional clarifications and support to schools participating in the pilot project. The feedback 
from the experiences of pilot schools could also be utilized to better inform the scaling-up 
process in coming years. With this caveat in mind, the following recommendations are pro- 
vided: 


e Consider establishing an ongoing monitoring protocol to identify the number of 
teachers from each school who may not be able to attend training sessions in the be- 
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ginning of the school year and encourage RESA staff and/or school leadership teams 
to provide supplemental training on an ongoing basis. 


e The 25 pilot schools would have benefited greatly if a single individual had been des- 
ignated from the Office of Information Systems at the WVDE to serve as a contact 
person and resolve technical issues in a timely manner for schools and districts. Go- 
ing forward, and with the addition of more than 100 schools in the pilot, the same 
option is not possible. Consider providing extensive training, specifically on the 
online system, to individuals either at the RESA or district level to serve as contact 
persons for their schools. 


e Consider making WVEIS on the Web accessible to educators outside of the school 
building. This may allow teachers to devote more time to various components of the 
system outside of the school day and avoid some technical difficulties resulting from 
limited bandwidth at the school level and, in the long-term, may lead to better overall 
quality implementation. 


e Consider making the self-assessment instrument available for the less experienced 
teachers and encouraging them to utilize the process for their own purposes. One of 
the most beneficial components of the system identified by respondents in the ad- 
vanced progression is the process of completing a self-assessment. 


e Consider providing onsite technical assistance to provide clarification on the process 
of setting student learning goals. All respondents identified this component as the 
most beneficial part of the system. Educators believe the process helps them ap- 
proach their teaching more systematically by helping them focus on areas of needed 
improvement for their students while it holds them accountable to the goals they 
have set for themselves and their students. At the same time, however, feedback from 
respondents indicates that participants still consider this process to be the most chal- 
lenging part of the pilot project. 


e Consider elucidating further the process for compiling additional evidence. Over a 
third of respondents indicated that they need additional clarification on this aspect of 
the pilot project. 
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